🚀 The docs preview is ready! Check it out here: https://modal-labs-examples--frontend-preview-dcedce0.modal.run

@pawalt looking forward to this one. I have a feeling that fine-tuning R1 would be great for us given its inference ROI.

nice! I think we'll keep the smaller model in the main example, since the download time and cold boots for a 70B model are pretty painful.

@pawalt have you looked into enabling tool calling by any chance?
https://github.com/vllm-project/vllm/blob/6d0e3d372446cde48b387d4d3530e25fc6e06320/vllm/entrypoints/openai/serving_chat.py#L47

i get back responses from the API when i add in enable_auto_tools and tool_parser params, but tool calling doesn't work that well. my guess is the granite parser doesn't work well for deepseek models - we'd need the tool template it was trained with. also tried llama3_json parser but that doesn't work either

api_server.chat = lambda s: OpenAIServingChat(
        engine,
        ...
        chat_template_content_format="raw",
        enable_auto_tools=True,
        tool_parser="granite",
    )

example response i get (should have used get_weather tool call):

{
  "id": "chatcmpl-52b3d25b480144ff9dfecebf8e739d0c",
  "object": "chat.completion",
  "created": 1737908244,
  "model": "DeepSeek-R1-Distill-Llama-70B",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "<think>\n\n</think>\n\nHi there! I suggest getting online to get real-time information. If you have any other questions, please don't hesitate to let me know!",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "stop",
      "stop_reason": null
    }
  ],
  ...
}

closing since we now have a DeepSeek-R1 example (cf #1057)